[P2P] Optimize block pool requester retry and peer pick up logic #170

yzang2019 · 2023-12-19T23:02:55Z

Describe your changes and provide context

Problems:
There are some inefficiency in the current p2p block sync module:

Even If a bpRequester has already downloaded the block from the peer, it can reset the block to nil whenever a retry is triggered. A retry can be triggered whenever a peer got disconnected or timedout, this could lead to a lot of extra network usages and block sync slow down. What usually happens is that during retry, it actually pick bad peers and give up previously downloaded block from good peers, then it end up having to wait for the response for a long time
Block sync reactor will need next two consecutive blocks to be available in the block pool in order to proceed the applyBlock, if any one is missing, the reactor will stuck and wait for the bprequester download to be finished
There's no timeout for a single block request, a block request can take more than a few minutes to get the response from a bad/slow peer, which would cause block sync to stuck during this time
We are not using the good peer list when picking available peers

Solution:

Introducing a retry reason, there are two major reasons, either we need to retry for a peer being removed due to disconnection, or we need to retry when we failed validating the block in the reactor.
When we handle the retry operation, we should check if we already have the block, if we already have the block read, we don't need to retry any more, unless the retry reason is bad block
Introducing an extra timeout for requester routine, when we wait for the block response, we should try a different peer if we haven't got the response after timeout

Testing performed to validate your change

Tested on standalone rpc nodes

philipsu522 · 2023-12-20T02:17:55Z

internal/blocksync/pool.go

+		if index >= len(goodPeers) {
+			index = len(goodPeers) - 1
+		}


nit - woudl this case ever be hit? could also use modulo to have more randomness

I thought math.randn is inclusive?

codecov · 2023-12-20T02:38:28Z

Codecov Report

Attention: 15 lines in your changes are missing coverage. Please review.

Comparison is base (72bb29c) 59.11% compared to head (6f4c6d3) 58.09%.
Report is 1 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #170      +/-   ##
==========================================
- Coverage   59.11%   58.09%   -1.03%     
==========================================
  Files         281      249      -32     
  Lines       38940    33907    -5033     
==========================================
- Hits        23021    19698    -3323     
+ Misses      14124    12645    -1479     
+ Partials     1795     1564     -231

Files	Coverage Δ
internal/eventbus/event_bus.go	`89.18% <100.00%> (ø)`
internal/p2p/peermanager.go	`80.14% <0.00%> (-0.29%)`	⬇️
internal/blocksync/pool.go	`77.96% <78.94%> (+0.49%)`	⬆️

... and 50 files with indirect coverage changes

* P2P Improvements: Fix block sync reactor and block pool retry logic

* Perf: Increase buffer size for pubsub server to boost performance (#167) * Increase buffer size for pubsub server * Add more timeout for test failure * Add more timeout * Fix test split scripts * Fix test split * Fix unit test * Unit test * Unit test * [P2P] Optimize block pool requester retry and peer pick up logic (#170) * P2P Improvements: Fix block sync reactor and block pool retry logic * Revert "Add event data to result event (#165)" (#176) This reverts commit 72bb29c. * Fix block sync auto restart not working as expected (#175) * Fix edge case for blocksync (#178) * fix evm pending nonce * fix test * deflake a test * de-flake test * Revert "merge main" This reverts commit 58b9424, reversing changes made to 02d1478. * consider keep-in-cache logic when removing from cache * undo test tweaks --------- Co-authored-by: Yiming Zang <[email protected]> Co-authored-by: Jeremy Wei <[email protected]>

* Standardize lag status response format (#187) * Standardize lag status response format * Fix flaky unit test * Make ReadMaxTxs atomic (#166) * Support pending transaction in mempool (#169) * fix unconfirmed tx to consider pending txs (#172) * fix pending pop (#173) * add TTL for pending txs (#174) * [EVM] Fix evm pending nonce (#179) * Perf: Increase buffer size for pubsub server to boost performance (#167) * Increase buffer size for pubsub server * Add more timeout for test failure * Add more timeout * Fix test split scripts * Fix test split * Fix unit test * Unit test * Unit test * [P2P] Optimize block pool requester retry and peer pick up logic (#170) * P2P Improvements: Fix block sync reactor and block pool retry logic * Revert "Add event data to result event (#165)" (#176) This reverts commit 72bb29c. * Fix block sync auto restart not working as expected (#175) * Fix edge case for blocksync (#178) * fix evm pending nonce * fix test * deflake a test * de-flake test * Revert "merge main" This reverts commit 58b9424, reversing changes made to 02d1478. * consider keep-in-cache logic when removing from cache * undo test tweaks --------- Co-authored-by: Yiming Zang <[email protected]> Co-authored-by: Jeremy Wei <[email protected]> * Fix bug when popping pending TXs (#188) * Add mempool metrics for number of pending tx and expired txs (#189) * Add metrics for mempool pending transaction size * Add expired tx count metrics * [EVM] Allow multiple txs from same account in a block (#190) * add mempool prioritization with evm nonce * fix priority stability * index fixes * replace with binary search insert * impl binary search * fix removeTx to push next queued evm tx (#191) * fix expire metric (#193) * [EVM] Fix duplicate evm txs from priority queue (#195) * debug duplicate evm tx * add more logs * add some \ns * more logs * fix swap check * add-lockable-reap-by-gas * add invariant checks * fix invariant parenthesis * fix log * remove invalid invariant * fix nonce ordering pain * handle ordering of insert * fix remove * cleanup * fix imports * cleanup * avoid getTransactionByHash(hash) panic due to index * use Key() to compare instead of pointer * [EVM] prevent duplicate txs from getting inserted (#196) * prevent duplicates in mempool * use timestamp in priority queue * [EVM] Add logging for expiration (#198) * add logging for expired txs * cleanup * [EVM] Avoid returning nil transactions on ForEach (#197) * remove heapIndex to avoid nil scenario * avoid returning nil in loop (mimic Peek) --------- Co-authored-by: Yiming Zang <[email protected]> Co-authored-by: codchen <[email protected]> Co-authored-by: Jeremy Wei <[email protected]>

* Perf: Increase buffer size for pubsub server to boost performance (#167) * Increase buffer size for pubsub server * Add more timeout for test failure * Add more timeout * Fix test split scripts * Fix test split * Fix unit test * Unit test * Unit test * [P2P] Optimize block pool requester retry and peer pick up logic (#170) * P2P Improvements: Fix block sync reactor and block pool retry logic * Revert "Add event data to result event (#165)" (#176) This reverts commit 72bb29c. * Fix block sync auto restart not working as expected (#175) * Fix edge case for blocksync (#178) * fix evm pending nonce * fix test * deflake a test * de-flake test * Revert "merge main" This reverts commit 58b9424, reversing changes made to 02d1478. * consider keep-in-cache logic when removing from cache * undo test tweaks --------- Co-authored-by: Yiming Zang <[email protected]> Co-authored-by: Jeremy Wei <[email protected]>

* Make ReadMaxTxs atomic (#166) * Support pending transaction in mempool (#169) * fix unconfirmed tx to consider pending txs (#172) * fix pending pop (#173) * add TTL for pending txs (#174) * [EVM] Fix evm pending nonce (#179) * Perf: Increase buffer size for pubsub server to boost performance (#167) * Increase buffer size for pubsub server * Add more timeout for test failure * Add more timeout * Fix test split scripts * Fix test split * Fix unit test * Unit test * Unit test * [P2P] Optimize block pool requester retry and peer pick up logic (#170) * P2P Improvements: Fix block sync reactor and block pool retry logic * Revert "Add event data to result event (#165)" (#176) This reverts commit 72bb29c. * Fix block sync auto restart not working as expected (#175) * Fix edge case for blocksync (#178) * fix evm pending nonce * fix test * deflake a test * de-flake test * Revert "merge main" This reverts commit 58b9424, reversing changes made to 02d1478. * consider keep-in-cache logic when removing from cache * undo test tweaks --------- Co-authored-by: Yiming Zang <[email protected]> Co-authored-by: Jeremy Wei <[email protected]> * Fix bug when popping pending TXs (#188) * Add mempool metrics for number of pending tx and expired txs (#189) * Add metrics for mempool pending transaction size * Add expired tx count metrics * [EVM] Allow multiple txs from same account in a block (#190) * add mempool prioritization with evm nonce * fix priority stability * index fixes * replace with binary search insert * impl binary search * fix removeTx to push next queued evm tx (#191) * fix expire metric (#193) * [EVM] Fix duplicate evm txs from priority queue (#195) * debug duplicate evm tx * add more logs * add some \ns * more logs * fix swap check * add-lockable-reap-by-gas * add invariant checks * fix invariant parenthesis * fix log * remove invalid invariant * fix nonce ordering pain * handle ordering of insert * fix remove * cleanup * fix imports * cleanup * avoid getTransactionByHash(hash) panic due to index * use Key() to compare instead of pointer * [EVM] prevent duplicate txs from getting inserted (#196) * prevent duplicates in mempool * use timestamp in priority queue * [EVM] Add logging for expiration (#198) * add logging for expired txs * cleanup * [EVM] Avoid returning nil transactions on ForEach (#197) * remove heapIndex to avoid nil scenario * avoid returning nil in loop (mimic Peek) * call callback from mempool (#200) * separate limit for pending tx (#202) * Add EVM txs eviction logic (#204) * Fix debug log (#205) * EVM transaction replacement (#206) (#208) * Add heapIndex with safety check (#213) * add heapIndex with safety check * cleanup * comment out for perf test * add back perf improvement * fix nil test * Use write-lock in (*TxPriorityQueue).ReapMax funcs (#209) ReapMaxBytesMaxGas and ReapMaxTxs funcs in TxPriorityQueue claim > Transactions returned are not removed from the mempool transaction > store or indexes. However, they use a priority queue to accomplish the claim > Transaction are retrieved in priority order. This is accomplished by popping all items out of the whole heap, and then pushing then back in sequentially. A copy of the heap cannot be obtained otherwise. Both of the mentioned functions use a read-lock (RLock) when doing this. This results in a potential scenario where multiple executions of the ReapMax can be started in parallel, and both would be popping items out of the priority queue. In practice, this can be abused by executing the `unconfirmed_txs` RPC call repeatedly. Based on our observations, running it multiple times per millisecond results in multiple threads picking it up at the same time. Such a scenario can be obtained via the WebSocket interface, and spamming `unconfirmed_txs` calls there. The behavior that happens is a `Panic in WSJSONRPC handler` when a queue item unexpectedly disappears for `mempool.(*TxPriorityQueue).Swap`. (`runtime error: index out of range [0] with length 0`) This can additionally lead to a `CONSENSUS FAILURE!!!` if the race condition occurs for `internal/consensus.(*State).finalizeCommit` when it tries to do `mempool.(*TxPriorityQueue).RemoveTx`, but the ReapMax has already removed all elements from the underlying heap. (`runtime error: index out of range [-1]`) This commit switches the lock type to a write-lock (Lock) to ensure no parallel modifications take place. This commit additionally updates the tests to allow parallel execution of the func calls in testing, as to prevent regressions (in case someone wants to downgrade the locks without considering the implications from the underlying heap usage). --------- Co-authored-by: Valters Jansons <[email protected]> * Pending Txs Update Condition (#214) * Add metrics for mempool size changes (#220) * [EVM] Adjust locking for replacement (#224) * Remove tx from cache when canAddPendingTx fails (#230) * add tx hash to evm info proto (#231) --------- Co-authored-by: codchen <[email protected]> Co-authored-by: Steven Landers <[email protected]> Co-authored-by: Yiming Zang <[email protected]> Co-authored-by: Jeremy Wei <[email protected]> Co-authored-by: Valters Jansons <[email protected]> Co-authored-by: Kartik Bhat <[email protected]>

* Perf: Increase buffer size for pubsub server to boost performance (#167) * Increase buffer size for pubsub server * Add more timeout for test failure * Add more timeout * Fix test split scripts * Fix test split * Fix unit test * Unit test * Unit test * [P2P] Optimize block pool requester retry and peer pick up logic (#170) * P2P Improvements: Fix block sync reactor and block pool retry logic * Revert "Add event data to result event (#165)" (#176) This reverts commit 72bb29c. * Fix block sync auto restart not working as expected (#175) * Fix edge case for blocksync (#178) * fix evm pending nonce * fix test * deflake a test * de-flake test * Revert "merge main" This reverts commit 58b9424, reversing changes made to 02d1478. * consider keep-in-cache logic when removing from cache * undo test tweaks --------- Co-authored-by: Yiming Zang <[email protected]> Co-authored-by: Jeremy Wei <[email protected]>

P2P Improvements: Fix block sync reactor and block pool retry logic

6bad60d

yzang2019 requested review from philipsu522, udpatil and Kbhat1 December 19, 2023 23:02

yzang2019 changed the title ~~P2P Improvements: Fix block sync reactor and block pool retry logic~~ P2P Improvements: Optimize block pool requester retry logic and peer pick up logic Dec 19, 2023

yzang2019 changed the title ~~P2P Improvements: Optimize block pool requester retry logic and peer pick up logic~~ P2P]: Optimize block pool requester retry and peer pick up logic Dec 19, 2023

yzang2019 changed the title ~~P2P]: Optimize block pool requester retry and peer pick up logic~~ [P2P] Optimize block pool requester retry and peer pick up logic Dec 19, 2023

yzang2019 added 5 commits December 19, 2023 15:22

Add flag for remove

fe91c46

Add ticker stop

f460671

Fix unit test

f43f8ca

Fix data race

8387cc1

Fix data race

011cbfa

philipsu522 approved these changes Dec 20, 2023

View reviewed changes

Fix lock usage

67cbe5f

Fix continue lable

6f4c6d3

yzang2019 merged commit 016c1b9 into main Dec 20, 2023
24 checks passed

yzang2019 added a commit that referenced this pull request Dec 20, 2023

[P2P] Optimize block pool requester retry and peer pick up logic (#170)

b4b8a3e

* P2P Improvements: Fix block sync reactor and block pool retry logic

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[P2P] Optimize block pool requester retry and peer pick up logic #170

[P2P] Optimize block pool requester retry and peer pick up logic #170

yzang2019 commented Dec 19, 2023 •

edited

Loading

philipsu522 Dec 20, 2023

yzang2019 Dec 20, 2023

codecov bot commented Dec 20, 2023 •

edited

Loading

[P2P] Optimize block pool requester retry and peer pick up logic #170

[P2P] Optimize block pool requester retry and peer pick up logic #170

Conversation

yzang2019 commented Dec 19, 2023 • edited Loading

Describe your changes and provide context

Testing performed to validate your change

philipsu522 Dec 20, 2023

Choose a reason for hiding this comment

yzang2019 Dec 20, 2023

Choose a reason for hiding this comment

codecov bot commented Dec 20, 2023 • edited Loading

Codecov Report

yzang2019 commented Dec 19, 2023 •

edited

Loading

codecov bot commented Dec 20, 2023 •

edited

Loading